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Background / Context: 

Description of prior research and its intellectual context. 

Students who do not develop adequate literaey skills by the end of elementary school are 
at higher risk of dropping out of school and face inferior labor-market options (National 
Governor’s Association, 2005; Vignoles, De Coulon, & Marcenaro-Gutierrez, 2011). Though 
fourth graders’ reading scores on the National Assessment of Educational Progress have been 
trending higher, Snow and Moje (2010) point out that score trends are flat among 8* and 12*'^ 
graders, (Lee, Grigg, & Donohue, 2007, p.3). These trends underscore the need for literacy 
support at the critical transition between elementary and secondary schooling (Chall & Jacobs, 
2003). 

One widely-used though under-evaluated method for improving student outcomes is 
through providing a “double dose” of instruction in subject areas tested for the purposes of 
NCLB, most notably, reading and mathematics. Recent evidence from Chicago suggests that 
these double-dose strategies with algebra instruction can have positive short-term impacts on 
student’s academic performance, as well as positive longer-run impacts on high-school 
graduation and post-secondary enrollment. (Nomi & Allensworth, 2009; Cortes, Goodman, & 
Nomi, 2012). However, little is known about the effectiveness of the double dose strategy for 
boosting literacy outcomes despite evidence that double-dose strategies have been and are used 
throughout the country (Cavanagh, 2006; Mazzolini & Morley, 2006; Paglin, 2003; Wanzek & 
Vaughn, 2008; Durham Public Schools, n.d). The paucity of good evidence on the effectiveness 
of literacy interventions at the crucial transition from elementary to middle grades is particularly 
notable in that the little evidence that exists is not causal. 


Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

In this paper provide causal evidence to answer the question, does the application of a 
double dose of literacy instruction in middle school improve student performance on subsequent 
academic outcomes. I focus on an intervention where the second dose of literacy instruction uses 
research-based instructional strategies, and I show that this instruction leads to systematic 
improvement in adolescent reading comprehension. I also consider how differences in the 
fidelity of implementation of this policy may impact the effects of the intervention both at the 
school-level and in aggregate. 

This district-designed literacy intervention used 5*-grade scores on the Iowa Test of 
Basic Skills to determine student eligibility for a double dose of literacy instruction in middle 
school. Students who scored at the 60* percentile or lower on the ITBS reading test were 
assigned, by rule, to be eligible for this literacy program. Using a regression-discontinuity design 
I found in early work that there was a positive aggregate effect of the policy for students who 
were just eligible for the intervention compared to those who were not. 

Setting: 

Description of the research location. 
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My study is situated in a large (roughly 90,000 students Pre-K through grade 12) 
suburban school district in the southeastern United States. The district is comprised of schools 
classified as traditional, charter, converted charter, and alternative schools. Both traditional and 
conversion charter schools are subject to district policies while alternative and other charters are 
exempt. I restrict my analysis to students who go through one of the 20 traditional or conversion 
charter middle schools that serve HCPS students in grades 6 through 8. My sample includes all 
students from the seven cohorts who took the 5* grade ITBS reading test in the school years 
2002-2003 through 2008-2009. The students I retain in my sample are representative of the 
students in both the traditional and conversion charter schools. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

HCPS resembles the changing demographic structure of many suburban settings, with 
substantial racial and socioeconomic variation. The student population is 43% white, 36% 
African American, 10% Latino/a, 8% Asian, and 3% other race. Forty-three percent of students 
receive free- or reduced-price lunch, 8% are English-Language Learners, and 18% have an 
Individualized Education Program. My analytic sample of roughly 7,000 students is drawn from 
a larger pool of over 30,000 students giving me statistical power to estimate modest effect sizes. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

Students in HCPS were assigned to receive supplementary reading instruction in middle 
school based on how they scored on the Iowa Test of Basic Skills (ITBS) in reading during their 
5* grade year. Students who scored below the nationally-defined 60**^ percentile on ITBS in 
reading were assigned, by rule, to complete the supplementary reading program in middle 
school. The HCPS policy was designed to enroll students in the supplementary reading course 
for all three (grades 6,7, and 8) years of middle school, with the goal of preparing students to 
meet proficiency requirements on the criterion-referenced 8* grade state test in reading (used in 
making decisions about grade promotion), and on the norm-referenced 8* grade administration 
of the ITBS in reading. 

The theory of change employed by HCPS is that enrolling students who have 
demonstrated a need for additional literacy support in a course that was designed to employ 
research-proven strategies is likely to improve literacy outcomes for those students. Specifically, 
this district drew on research from Dole, Duffy, Roehler, and Pearson (1991), and designed the 
supplementary reading class to explicitly dwell on seven “basic” reading strategies: activating 
background knowledge, questioning the text, drawing inferences, determining importance, 
creating mental images, repairing understanding when meaning breaks down, and synthesizing 
information. In addition, the district also encouraged the use of writing activities to support each 
of these seven reading strategies. 

Though the research from Dole and colleagues is more than twenty years old, more 
recent research continues to substantiate the use of these strategies, particularly with adolescents. 
A meta-analysis on the effectiveness of reading interventions for struggling readers in grades six 
through twelve revealed that many of the same strategies suggested by Dole and colleagues were 
used across the thirteen studies that could be included in that meta-analysis (Edmonds, Vaughn, 
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Wexler, Reutebuch, Cable, Klinger Taekett, Sehnakenberg, 2009). This meta-analysis found a 
large effeet size of 0.89 SD for reading eomprehension outeomes. Evidenee from another reeent 
meta-analysis on writing to read, further supports the strategies employed by HCPS. Graham & 
Hebert (2012) found that writing to read strategies improve student reading eomprehension by 
about 0.37 SD. In yet another teaeher-delivered intervention, Vaughn, Klingner, Swanson, 
Boardman, Roberts, Mohammed, & Stillman-Spisak (2011) performed an experimental 
evaluation of collaborative strategic reading (CSR) with middle school students, where English- 
language arts teachers provided a multicomponent reading comprehension instruction twice a 
week for 18 weeks, and found modest positive effects on reading comprehension. All of this 
more recent evidence suggests that the research used to design the supplementary reading class 
continues to be valid and relevant. 

Research Design: 

Description of the research design. 

In the district, student assignment to the supplemental reading class was made using a 
cutoff rule based on a student’s 5* grade test score, allowing me to use a regression-discontinuity 
approach to obtain an unbiased estimate of the causal impact of the intervention on student 
outcomes. The student’s position relative to this cutoff provided an indicator of intent-to-treat, 
which I used to instrument for their “take-up” of the supplementary reading intervention. Thus, I 
was able to identify the causal impact of enrollment in the program for students near, but on 
opposite sides of the cutoff. I employ a two-stage least squares approach where a student’s 5**^- 
grade ITBS reading score and a dichotomous indicator of receiving an offer of supplementary 
predict the exogenous portion of enrollment in the intervention. I then use the fitted probabilities 
of enrolling from the first-stage to predict the impact of enrollment on later outcomes. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

My data are drawn from a comprehensive administrative data set covering all students 
enrolled in the district during the school years of 1999-2000 through 2009-2010. This dataset 
contains test scores and enrollment data for students in middle school and follows them 
longitudinally within the district into high school. The data include course enrollment data, 
mandated state accountability test scores in reading, literature and mathematics, ITBS scores 
from grades five and eight, high school end-of-course examinations, and SAT scores. 

Findings / Results: 

Description of the main findings with specific details. 

I find that there was a positive and statistically significant improvement in student 
reading state test scores in 6* grade, for students in the immediate vicinity of the cut-off, with 
smaller effects in 7* grade, and complete fade out of the effect by 8* grade. In addition, I find 
that the intervention had a small positive impact on student percentile rank on the Iowa Test of 
Basic Skills in grade 8, indicating that the measured learning gains suggest real learning and not 
an artifact of potential teaching to the test (see Eigure Al). 
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Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

My findings suggest that a research-based supplementary reading course in middle school 
can, on average, boost short-term measures of student’s reading comprehension. These findings 
are particularly impressive in that they were achieved by deploying this “double dose” strategy 
across twenty individual middle schools, and taught by more than twenty individual teachers. 
These aggregate effects can, however, mask the potential for heterogeneity in the delivery of this 
intervention. In particular, the ability to provide a strong evaluation under this policy design may 
suggest that adopting assignment rules when deciding who to assign to support courses could be 
fruitful in other school districts (Schlotter, Schwerdt, &Woessmann, 2011). Such rules allow for 
the estimation of causal estimates and can reduce the continuation of ineffective program, and 
help districts develop or switch to interventions that have been proven to be effective. 

These benefits are not without several potential pitfalls. Districts and researchers must 
responsibly examine the potential for mixed effects across individual school settings, to ensure 
both the proper fidelity of implementation, as well as to examine potentially differential degrees 
of success with the policy. In addition, changing the threshold used to make assignment to 
supplementary courses can allow for the examination of the effectiveness of the program at 
different margins, but no assumption should be made that all margins will be similarly impacted. 
Cutoff scores must be chosen, and interventions designed, in such a way as to be consistent with 
the needs of the population it is intended to impact. In HCPS, all students scoring below the 60* 
percentile were eligible for supplementary reading however, this evaluation only addresses the 
impact of those who were eligible but near the cutoff. Other means of evaluation, and perhaps 
interventions tailored to learners who scored in lower percentiles, is necessary if we are to 
achieve equitable outcomes in education. It bears further note that simply because a program is 
successful on one margin, it need not necessarily maintain its impact when extended to students 
on other margins and of different abilities. 

Further research into “double dose” literacy interventions like the one in HCPS is 
warranted. This research should have the goal of collecting data on additional cohorts of 
students, and following those already represented in the data over longer time periods so that we 
may learn whether there are longer-term impacts of the supplementary reading course; such as on 
SAT scores, high school graduation, or decisions to apply to or attend college. Establishing the 
effectiveness of similar supplementary coursework should be pursued in other research contexts 
as well, since our ultimate concern is with long-term outcomes that we believe are associated 
with measures of adolescent literacy. 
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Figure 3; Fitted effeets of an additional semester of treatment on 6*-grade reading and 8*-grade 
ITBS pereentile overlaid on evidenee of diseontinuity in exposure to treatment. 

Panel A: 6*-grade Reading 



Panel B: 8^*’-grade ITBS Pereentile 
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